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ABSTRACT 



This paper reports on a pilot project in Asia sponsored by 
the IFLA (International Federation of Library Associations and institutions) 
Round Table of Library and Information Science Journals that had the 
following objectives: to identify the most commonly accepted criteria for 
evaluating LIS (Library and Information Science) journals; to evaluate the 
success of LIS journals in meeting the criteria; to suggest critical success 
factors for improving the quality of LIS ' j ournals ; and to assess the 
feasibility of an international project aimed at determining LIS journal 
quality and success factors. Twenty LIS journal editors in Australia, China, 
Japan, Malaysia, and the Philippines were interviewed. Quality of papers 
emerged as the Key category, with a number of criteria used to determine this 
quality, including advancement of knowledge, new information or data, 
theoretical soundness, level of scholarship, acceptable research design, and 
appropriate methodology and analysis. In-depth analysis of sample copies of 
journals provided by the editors was performed, and the quality criteria were 
ranked by importance. It was found that, given the relative ease with which 
data were collected for the pilot study and the ready participation of most 
editors approached for input, a more substantial project would be possible 
with only low level funding. (Contains 11 references.) (MES) 
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Abstract v 

Reports on a project sponsored by the IFLA RTLISJ aimed at determining factors 
that contribute to journal quality. It is a pilot project in Asia with the aim in part 
of determining whether a larger international project might be feasible. Quality 
of papers emerged as the key category, with a number of criteria used to 
determine this quality. It was found that, given the relative ease with which data 
were collected for the pilot study and the ready participation of most editors 
approached for input, a more substantial project would be possible with only low 
level funding. 



Paper 

Introduction 

Journal 'quality' traditionally has been measured against quantitative measures 
such as circulation, total number of pages per volume, number of times cited in 
the literature, coverage by indexing services. This pilot research project, 
undertaken on behalf of, and partly funded by, IFLA's Round Table of Library 
and Information Science Journals (RTLISJ), follows a presentation by one of the 
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investigators at IFLA Bangkok which sought to establish general criteria for 
assessing journal submissions from Asian authors (Gorman 1999) That paper 
suggested that quantitative measures were perhaps unsuitable for evaluating the 
qualitative factors that contribute to journal excellence. Following that 
presentation, and picking up a suggestion made some years earlier by Maurice 
Line, Ludmila Kozlova, Chair of RTLISJ, asked the presenter to undertake a pilot 
study of journal quality. 

This investigation took a more qualitative approach to understanding journal 
quality based on perceptions of key stakeholders - the editors, referees, editorial 
board members and impartial readers. The specific target of the study was Library 
and Information Science (LIS) journals published in Asia. This region was chosen 
partly because of investigator familiarity and ready access - both are editors of 
journals based in Asia. It is also an area in which LIS journal publishing is well 
established - there are more than 200 LIS journals in China alone (Gong and 
Gorman 2000), and even such small areas as Singapore and Hong Kong have 
major journals in this field. Furthermore, many of the journals are published in 
less developed countries, with information infrastructures that are often fragile, 
under-funded and ignored by government. If the pilot study contributed to a more 
robust infrastructure (of which LIS journals are certainly a part), this would be a 
bonus. 

Essentially, however, this study was meant to be a pilot project for a more broadly 
based international investigation of journal quality. There is a widespread view 
that LIS journals around the world are of somewhat uneven quality, that they do 
not meet a common set of standards of excellence, that there may not even be 
such a set of standards, that journals in developing countries in particular might 
benefit from better understanding of quality, and that all such journals can 
become more effective channels for the communication of theory and practice to 
the various information professions within LIS. Improved journal quality might 
provide a more effective channel through which ideas, theories, practices and 
news can be communicated between practitioners, educators, vendors and other 
interested parties. 

However, 'improved journal quality' must take into account the reality that 
contributors to journals write for a variety of reasons (Gorman 1999). Most 
commonly, we write to disseminate new research findings or ideas. The 
publication of a paper establishes precedents in the formation of new knowledge, 
and it puts the new information in the professional domain where it can be 
scrutinised, criticised and either accepted or rejected. It may then contribute to 
further discourse. The author also makes personal gains by adding to a list of 
publications that can be used for tenure and promotion, for gaining professional 
acceptance that may lead to speaking engagements, consultancy work, perhaps 
even awards. 

There is, then, an apparent contradiction between the intrinsic and extrinsic 
reasons for scholarly and professional writing, for personal benefits can result 
from numerous papers of indifferent quality, whereas the imperative for the 
discipline is the discovery of new conceptual approaches and new techniques, for 
which the need is for papers of the highest quality. That the scholarly 
communication system has survived almost unchanged for so long shows its 
robustness, but the inherent contradictions in the system make it vulnerable to 
distortion under certain circumstances. If, for example, the 'publish or perish' 
imperative creates such demand among hopeful authors that editors are 
overwhelmed with manuscripts of an indifferent quality, then there is potential for 
the erosion of standards. This may occur if new journals start up to cater for the 
unfulfilled demand from hopeful authors. There is some emerging evidence, from 
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current research by Philip Calvert and Shi Zengshi on quality and quantity in 
journal publishing, that this has happened in China already. 

Both extrinsic and intrinsic reasons lead to publications that might be assessed by 
quantitative means, but also by qualitative means. The fact that paper x is cited y 
times is not an indication of quality, but rather that it is cited - it is available, it is 
in a journal held by many libraries, the author (or publisher or editor) is 
particularly good at self-promotion. Behind the quantifiable factors, then, are as 
yet untested qualitative factors, which is what has led to the present project. 



Related Literature 

Most assessments of journal quality are driven by the practical necessity of 
producing ranked lists of journals in each academic discipline so that tenure and 
promotion committees can assess the publication lists from applicants. One result 
of this is discussed by Ali, Young and Ali (1996). They indicated that core lists 
have been drawn up for many disciplines, and that these are usually based on 
citation analysis, circulation figures and coverage in indexing and abstracting 
services. Altmann and Gorman (1999), however, have cast doubt on the efficacy 
of journal citedness as a criterion of value with relation to acquisitions and 
relegation, and the same might be said of citedness as a criterion for developing 
core lists. 

An intriguing project was conducted by Chressanthis and Chressanthis (1993), 
one an economist, the other a serials librarian. They hypothesised that journal 
quality was dependent upon a number of cost factors, and especially the 
manuscript submission fee. They found that their definition of journal quality 
most closely correlated with the total number of journal article pages printed in 
the year, journal age, the editor's institutional affiliation, the manuscript 
submission fee, and the total circulation of the journal. There was insignificant 
correlation with the presence of advertising and the journal affiliation with a 
professional association. The measure of quality used by the authors, however, 
was the total number of citations to the journal: therefore, independent variables 
such as the number of articles printed and the total circulation of the journal 
would inevitably have a high correlation with this measure of 'quality'. Citation 
analysis will always be biased in favour of high circulation journals, which is why 
even Garfield (1977) cautioned against using it on its own as a measure of journal 
quality. 

Anderson (1997) has provided another view of journal quality, arguing that it is 
linked to 'excessive publication' - submitting the same paper to two or more 
journals. His research was mainly qualitative in that he used journal editors' 
opinions on a number of questions as his major source of information - the same 
method employed in the present investigation. Editors believed their guidelines to 
authors gave some protection against double publication of papers, though a 
sizeable number admitted that their guidelines needed review and strengthening. 
As a sidelight, Anderson's qualitative research suggested that editors have 
primary responsibility for quality, referees the next most influence, with the 
editorial boards a distant third. 

In a lengthy, thorough and original paper on journal quality, Day and Peter (1994) 
used qualitative methods to ask subscribers and authors, plus some editors and 
editorial advisors, what they thought about journal quality. From the LIS 
literature they created a list of quality criteria to use in their research: 
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• research design 

• depth 

• purpose 

• practical examples 

• rigour 

• descriptive versus analytical 

• presentation 

• conclusions 

• relevance 

• focus versus generalisation 

• application 

• relevant references and recency 

• clarity of thought 

• structure 

• interest. 

Reviewers added additional criteria to the authors' original lists, asking the 
following questions: 

Does the article add to what is already known? 

Is the article demonstrably related to what has previously been written? 

Are the arguments employed valid in terms of the body of knowledge? 

Is the article easy to read? 

Do the arguments flow logically? 

Does the article make a difference? 

Are the conclusions strong? 

From all of this a number of criteria were selected as most relevant and applicable 
without the need for extensive explanation as to meaning, and these criteria were 
seen to fall into three major categories: 

• prestige (of the editor, etc.) 

• properties of articles within a journal (e.g. methodological rigour) 

• presentational aspects. 

This pilot project focussed on the second of the larger categories, because the 
investigators felt this was likely to be the most crucial in determining journal 
quality. The other categories were also included in the broad investigation, but 
only the disaggregated properties of the journal articles were examined in detail. 

Day and Peter's qualitative approach is echoed to an extent by Nkereuwem 
(1997), who used Lester's method of journal evaluation that combines input 
measures, decision measures and output measures into an index of journal quality 
for ranking journals. There is a bias in this method which favours journals with a 
wide market reach. No doubt some quality journals are familiar to LIS 
professionals because of their intrinsic quality, but familiarity can also be 
achieved by 'free' distribution to association members or by low subscription 
rates, and this hardly equates with quality. This is often the case in developing 
countries, where even quite 'flimsy' LIS journals are well known because of their 
wide distribution. 



Objectives and Methods 

The pilot study had four principal objectives: 
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• to identify the most commonly accepted criteria for evaluating LIS joumais 

• to evaluate the success of LIS joumais in meeting the criteria 

• to suggest critical success factors for improving the quality of LIS joumais 

• to assess the feasibility of an international project aimed at determining LIS 
journal quality and success factors 

These objectives were achieved through three inter-related activities. First, other 
writings were reviewed in order to develop tentative criteria of journal quality. 
Second, the investigators interviewed selected LIS journal editors in Asia, asking 
for their views on the factors that contribute to journal quality. Third, selected 
Asian LIS joumais, specifically those edited by the editors interviewed, were 
examined by the investigators to see how closely they match the standards 
expressed by the editors during the interviews. Fourth, the investigators combined 
the finding of steps 1-3 to produce a list of common criteria of journal quality that 
might be used in a fully international project. 

In reviewing the LIS literature for relevant previous writing on journal quality it 
became clear that most authors in this field have used quantitative methods. 
Nevertheless, their criteria could be used in a qualitative study, so they were 
drawn together and collated into groups of like categories. The resulting main 
categories were: 

• qualities of the papers 

• presentation factors 

• aspects of prestige 

• income factors 

The first three categories were derived from Day and Peter (1994), while the 
minor category of income factors was taken from Chressanthis and Chressanthis 
(1993). It became clear early in the process of gathering data from Asian editors 
that the qualities of the papers mattered far more than the other categories, so the 
investigation focused on the criteria in that category. 

Twenty LIS journal editors in Australia, China, Japan, Malaysia and the 
Philippines were approached. Some were interviewed in person, and others were 
asked to write brief notes on how articles were chosen for publication, and how 
they as editors ensured that the selected articles matched the aims and purpose of 
their joumais. They were also asked to provide, in keywords, the criteria they 
used for selection or rejection for publication of a contribution, and to rank their 
criteria in order if they felt it possible to do so. 

The initial list of criteria concerning qualities of papers was edited as a result of 
input by the various editors. Gorman (1999) had suggested six criteria for the 
evaluation of submissions to Asian LIS joumais, and it was these six that the 
editors agreed ultimately accepted as most relevant to the assessment of paper 
quality/content: 

• advancement of knowledge 

• new information or data 

• theoretical soundness 

• level of scholarship 

• acceptable research design 

• appropriate methodology and analysis. 

These six criteria were used for the next stage of the project, which involved 
in-depth analysis of the sample copies supplied by the editors. Papers were 
chosen at random for examination, five papers from each journal being the norm. 
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Once a paper had been read, a 'score' was given (from 0 to 10, with 10 being the 
highest standard) based on how well it had measured against each specific 
criterion. The purpose was not to assess the journals, but rather to compare the 
criteria, so an average score was calculated for each of the six criteria, and then 
they were placed in a ranked list. The two highest rated categories (new 
information or data, acceptable research design) scored well ahead of the others. 
The third category (level of scholarship) was on its own in third place, with the 
other three grouped together at the bottom. From this analysis, the criteria can be 
placed in descending order of importance as follows: 

• new information or data 

• acceptable research design 

• level of scholarship 

• theoretical soundness 

• advancement of knowledge 

• appropriate methodology and analysis. 



Analysis of the Data Collected from Asian Journals 

Quality of papers 

Many editors said that they looked for new ideas and innovative thinking in a 
paper. For some this was the top criterion. The rationale appears to be something 
like this: editors need to publish a journal that attracts subscribers, both 
institutional and individual. One way to do this is to include articles that seem to 
be relevant to the common experience of practitioners who, in LIS at least, form 
the majority of the potential market. Editors sometimes expressed this directly by 
saying they wanted their journals to be 'practical' rather than theoretical. 
Practitioners, however, do not want to be told how to carry out their jobs as they 
are at present; rather, they seek new and more efficient and effective ways to 
work. This is why editors value 'newness' in manuscript content above other 
criteria. For one editor this was closely related to his role perception. He felt he 
had to be a 'missionary editor' (his words), trying to persuade librarians to reflect 
on their roles and to think in terms broader than the next crisis. 

Another highly ranked criterion was finding manuscripts that matched the scope 
and purpose of the journal itself. For journals with a clear niche in the market this 
is easy enough to understand; however, as many journals seem to accept papers 
on almost any topic within LIS, despite stated criteria which are more specific, it 
is not easy to accept that this criterion has much real value overall. Perhaps it is 
more a matter of matching the intended market of the journal, and this means 
practitioners at a certain level of education, experience, interests, and so on. 

An alternative view of the importance of a paper's subject content was put by one 
editor when he indicated that he sought manuscripts with 'significance in the area 
concerned' - meaning that he wanted to publish articles that made a real 
contribution to the discourse in the paper's chosen area. This could be interpreted 
as being the paper's 'impact', a quality that is difficult to ascertain with any degree 
of accuracy. As noted earlier, the impact of a paper is often measured by the 
number of citations it attracts; a recent Asian example of the method of measuring 
a journal's impact by citations is Sen (1999). Other editors wanted to publish 
articles with more research content (one suggested College & Research Libraries 
as a model to follow), but most said they did not receive enough manuscripts 
containing sound research. 
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An impartial assessment by the investigators of papers in the submitted journal 
issues suggests that editors are doing well overall at finding papers that match 
their own selection criteria. The highest average score for the papers was for the 
criterion, new information or data, and most editors had said this was their top 
priority. Acceptable research design and level of scholarship also scored well. The 
lowest average score given to the randomly chosen articles was for appropriate 
methodology and analysis. This is a worrying result, for if the selection of papers 
is skewed towards newness at the cost of accepting contributions with a dubious 
methodology, then quality is quickly eroded. In fairness to the editors and their 
journals, it must be made plain that no obvious misinformation or dubious theory 
was found in the course of this investigation, and the concerns expressed here are 
only about what might occur. 

Prestige 

Chressanthis and Chressanthis (1993) claimed that the affiliation of the journal 
had little impact on perceptions of its quality. This may be true of Western 
countries, but possibly not so in Asia and developing countries in other parts of 
the non-Westem world. It seemed that the sponsoring institution played a 
significant part in establishing the status of an LIS journal in China, with the three 
highest ranked journals being sponsored by the National Library, the Chinese 
Academy of Sciences and Peking University. The same was true of journals with 
institutional or association affiliation in the other countries of the pilot study, 
including Australia. It should be noted, however, that only one journal in the 
study was a commercial undertaking, and the sample would be very different in 
this respect if the project were expanded to Europe and North America. Generally 
this suggests that in developing countries it would be beneficial to establish LIS 
journals with the backing of government departments, prestigious research 
institutes or well-respected professional bodies. The journal's age also appeared to 
be significant in determining prestige, with older journals viewed as more 
prestigious by most editors. 

Presentation 

It is difficult to discuss presentational aspects of journal publication simply 
because it is a highly subjective category, and few editors or readers claim 
expertise in matters of design. It is much more likely to be the publisher who has 
specific knowledge in this area, so perhaps a fuller project might include 
publishers among the key stakeholders. Nevertheless, it is clear from the pilot 
study that editors and their boards feel intuitively that presentation is a factor in 
increasing subscriber perceptions of their journals' quality. Literally all of the 
examined Chinese journals were using a cover design introduced within the past 
two years, or were using a higher grade paper than in the past, and all Chinese 
editors seemed conscious of layout, font and other design issues in ways that 
would not have troubled them even a short time ago. Several editors, however, 
when questioned about their technical knowledge, admitted that they felt uneasy 
about commenting on matters related to physical presentation and style. As one 
editor said, 'I think I know what I like and what looks good, but this is a very 
personal matter and I could not give any objective assessment of what a journal 
ought to look like.' Matters of presentation, then, are very much in the eye of the 
beholder- not a satisfactory situation when seeking to determine quality in an 
objective manner. 



Recommendations for Improving Journal Quality in Asia 
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Running through the data on journal quality, prestige and presentation is a very 
clear signal that the editor's role is critical. Some sponsoring institutions closely 
associated with research and scholarship appreciate the importance of selecting a 
well-qualified editor, and in developed countries with a larger number of 
commercially published journals the selection of editors is an increasingly 
rigorous process. However, not all journals have the benefit of this sort of 
support, certainly not in developing countries. In many cases the editor is a 
willing volunteer with few credentials and no training for the job. The more 
successful editors in Asia - that is, editors whose journals seem to set high 
standards and achieve them - already hold higher degrees in LIS or related 
disciplines. In our view improving the educational standard of other editors would 
almost certainly have a positive impact on journal quality. 

Similarly, the editorial board needs to represent a good spread of specific 
knowledge in the discipline, including an awareness of overseas developments. 
The editors of some journals admit that board members are chosen primarily on 
the basis of 'cronyism'; these journals tend to fare less well in our assessment of 
journal quality. The message here is simple - editors need to select board 
members who are well qualified, and who together represent the breadth and 
depth desired in the journal content. 

Countries whose LIS journals are primarily in a language other than English 
would benefit from the assistance of anglophone journals in promoting their 
content to the widest possible audience and to get them into international 
discourse. One suggestion for improving access to a wider audience is for a 
developing country journal to seek overseas alliances, such as exchange 
agreements. This would result in a paper being published in the 'local' literature to 
the benefit of the profession in that country, but equally the best of the local 
literature would also be published in a leading international journal, thus allowing 
its ideas and information will enter professional discourse with greater speed and 
certainty. This kind of exchange agreement exists in a few cases; for example, 
College & Research Libraries exchanges one paper per year with Journal of 
Library Science in China, and Online Information Review is currently exploring a 
similar relationship with journals in China and Thailand. 

For the most part, however, established journals in North America especially are 
somewhat parochial in their content - an exception is Library Collections, 
Acquisitions and Technical Services, which is so remarkable in this respect that it 
has figured in a review of the journal (Nisonger 1996); British and European 
journals do rather better, and it would be worth developing country journals 
exploring the possibility of article exchange agreements with titles in these 
regions. 

Several editors said they received manuscripts 'out of field' which were rejected 
almost automatically. This suggests that the solicitation process might lack focus, 
yet almost all journals publish guidelines or information for authors. Often, 
however, these seem to provide guidance only on formatting of the manuscript, 
preparation of the copy, how many copies to provide, footnotes style, and so on. 
Few mention methodological rigour, use of the literature or the required depth of 
analysis. Several editors agreed that their guidelines needed revision and 
strengthening. In addition more could be done to disseminate the guidelines in 
order to attract better manuscripts that are clearly within a journal's scope. Many 
Asian journals already use their Web sites to publicise author guidelines and 
recent tables of contents, and this seems to be a simple and relatively inexpensive 
way of telling prospective authors what the editors want to see submitted - as long 
as the guidelines actually address substantive matters of content. Workshops held 
in conjunction with professional conferences are another means used by editors to 
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inform potential authors of what is expected in an acceptable submission. 

Unfortunately, existing guidelines are only a weak defence against unscrupulous 
authors. Many editors said that they uncover several cases annually of papers 
being submitted to more than one journal. Obviously many more cases escape 
unnoticed. Whether this severely affects journal quality, as Anderson (1997) 
claims it does, was not something this pilot project was able to determine. 

Clearly, though, the 'publish or perish' imperative in some Asian countries leads a 
few desperate and unethical individuals to try such tactics, and editors need better 
protection than they have at the moment. 



Recommendations Regarding a Wider Project 

In addition to assessing the quality of LIS journals in Asia, this pilot project was 
designed to test how well an investigation into LIS journal quality could work, 
whether qualitative measures could be applied and whether a wider project might 
be feasible. There is a lack of literature on qualitative methods used to assess 
journal quality, so new methods of assessment have to be designed. The pilot 
project has shown that this can be done using criteria from the literature, 
supplemented with new criteria suggested by experienced editors and readers. 
Criteria established during the pilot project offer an acceptable starting point for a 
wider project, and it is assumed that these criteria will require only fine-tuning in 
a larger project. 

How much larger? The pilot study collected data from 20 journals; the full project 
should expand this to 200-300 journals from around the world. These journals 
might be divided geographically as follows: 

12 Asian countries 

10 European countries 

2 North American countries 

10 Latin American and Caribbean countries 

8 Africa countries 

4 Middle Eastern countries. 

A project of this magnitude requires the willing collaboration of editors, as well 
as some support from readers of the journals. The pilot study has shown that 
journal editors are surprisingly willing to co-operate with this sort of research, 
and there should be no difficulty attracting more volunteers for a larger project. 
Not must editors be willing to support the project, but the team itself needs to be 
fairly large in order to cope with some of the languages in which many journals 
are published, including Chinese, Japanese, Arabic and Russian. In addition it 
will be important to involve publishers and related groups (graphic designers, for 
example) in the project so that matters of presentation can be addressed more 
adequately than in the pilot. Up to 10 investigators would need to be managed by 
the principal investigator. 

All of this in our view could be managed on a budget of about US$6000. Is it 
worth the investment? 
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